Improve multi_field syntax #4521

clintongormley · 2013-12-19T16:42:27Z

Proposed syntax for multi_field

Any field type (other than object, nested) should accept a fields parameter which defines any extra fields that should be indexed. So for example a simple field like:

{
    "title": {
        "type": "string"
}}

can be converted into a multi-field by adding a fields parameter:

{
    "title": {
        "type": "string",
        "fields": {
            "raw": { "type": "string", "index": "not_analyzed" }
       }
}}

The text was updated successfully, but these errors were encountered:

…hat can be set on any core field. When upgrading to ES 1.0 the existing mappings with a multi-field type automatically get replaced to a core field with the new `fields` opt Closes to elastic#4521

…hat can be set on any core field. When upgrading to ES 1.0 the existing mappings with a multi-field type automatically get replaced to a core field with the new `fields` option. If a `multi_field` type-ed field doesn't have a main / default field, a default field will be chosen for the multi fields syntax. The new main field type will be equal to the first `multi_field` fields' field or type string if no fields have been configured for the `multi_field` field and in both cases the default index will not be indexed (`index=no` is set on the default field). If a `multi_field` typed field has a default field, that field will replace the `multi_field` typed field. Closes to #4521

clintongormley · 2014-01-14T20:55:34Z

Moved the path part of the issue to to #4729

martijnvg · 2014-01-14T20:57:08Z

Pushed to master via: 943b626

…hat can be set on any core field. When upgrading to ES 1.0 the existing mappings with a multi-field type automatically get replaced to a core field with the new `fields` option. If a `multi_field` type-ed field doesn't have a main / default field, a default field will be chosen for the multi fields syntax. The new main field type will be equal to the first `multi_field` fields' field or type string if no fields have been configured for the `multi_field` field and in both cases the default index will not be indexed (`index=no` is set on the default field). If a `multi_field` typed field has a default field, that field will replace the `multi_field` typed field. Closes to elastic#4521

apatrida · 2014-01-22T08:20:28Z

It isn't clear if the settings inheritance works, for example if "type" is left out of the sub-field, it produces an error:

{"error":"MapperParsingException[mapping [default]]; nested: MapperParsingException[No type specified for property [stopped_fuzzy_metaphone]]; ","status":400}

So does that work, or just not for type (therefore the sample is wrong). What settings inherit, which do not?

martijnvg · 2014-01-22T08:48:15Z

@jaysonminard The settings inheriting didn't get in. For some reason I forgot to update the issue description and documentation. I'll update this now.

apatrida · 2014-01-22T09:03:43Z

@martijnvg

Ok, but the default field being from the root still works ok? For example:

{
    "title": {
        "type": "string",
        "fields": {
            "raw": { "type": "string", "index": "not_analyzed" }
       }
  }
}

produces title and title.raw?

martijnvg · 2014-01-22T09:05:36Z

Yes, that produces a title and a title.raw field.

apatrida · 2014-01-22T09:39:43Z

ok, with a field pushing content to one of these (a source field using copy_to into a dest multi-field), I am not getting the expected results, will try a small experiment to see more specifically what is going on.

martijnvg · 2014-01-22T09:42:47Z

If your document looks like this:

{
    "title" : "my title"
}

and using the above mapping snippet should yield two fields a title and title.raw field.

apatrida · 2014-01-22T09:48:05Z

either copy_to doesn't work, or it doesn't work in conjunction with this. 1 moment for example

apatrida · 2014-01-22T09:57:42Z

ok, don't know where I got the idea that copy_to was in 1.0.0.RC1, maybe only on master (the docs are only for 0.90 branch or master, there are no 1.0.0.RC1 specific docs?). That is the cause of my problem, using something allowed by mappings parser, but doesn't actually work. I'll go back to full names and using path variable to make the copies.

"title": {
            "type": "string",
            "index": "no",
            "store": "no",
            "copy_to" : [ "search_title", "search_combined", "other_title" ],
            "fielddata": {
                "format": "disabled"
            }
        },

...
"search_title": {
            "type": "string",
            "index": "analyzed",
            "fields": {
                "raw": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        },

martijnvg · 2014-01-22T10:00:27Z

Yes, copy_to isn't in 1.0.0.RC1 and there are no rc1 specific docs... there will be specific docs for 1.0 when 1.0.0.GA gets out.

apatrida · 2014-01-22T10:02:51Z

Ok, that makes it hard to test 1.0.0.RC1 with 1.0.0.RC1 features. The release candidate isn't the main download link, has no docs, and seems like it could use more attention getting markers so that it is heavily tested before GA.

By the way, this link related to this actual issue (multi fields docs) on master is broken:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/multi-fields.html

(linked to from: http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/mapping.html)

So there is not current multi-field docs that I can find.

martijnvg · 2014-01-22T10:05:58Z

I just removed that specific link from the documentation and added the multi fields documentation to the core types page: http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/mapping-core-types.html#_multi_fields_3

I guess this must be some browser cache, that showed the old link.

martijnvg · 2014-01-22T10:07:54Z

Also I'll annotate the multi fields and copy to with the right version, in order to avoid confusion.

apatrida · 2014-01-22T10:14:16Z

Ok. Without copy_to there is no way to do what I wanted. Path of "just_name" affects all 'children' of the mutli-field, when I only want one of them to go to the just_name and the rest to be suffixes. path only applies at the top level of the field yes, and can't change for just one of the multi-values?

or can the name include a period. so that TITLE could go to TITLE, TITLE.NOSTOP, TITLE.EXACT, and ALL_TITLES at the same time without changing the input JSON.

martijnvg · 2014-01-22T10:43:27Z

Yes, path applies to all multi fields of a core field, but you can define another fields inside another fields. I think that something like this will result into something that you want:

curl -XPUT "http://localhost:9200/test1" -d'
{
    "mappings": {
        "test1" : {
            "properties": {
                "title" : {
                    "type": "string",
                    "fields": {
                        "raw" : {
                            "type": "string"
                        },
                        "search_title" : {
                            "type" : "string",
                            "index": "no", 
                            "path": "just_name",
                            "fields" : {
                                "x" : {
                                    "type": "string",
                                    "index_name": "search_title"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}'

But this way more verbose and less readable than when copy_to is used.

apatrida · 2014-01-22T10:46:55Z

Ok, the default field doesn't actually work right. (not related to copy_to)

 "content_cleaned": {
                    "type": "string",
                    "index": "analyzed",
                    "analyzer": "SearchWithSpecials",
                    "store": false,
                    "norms.enabled": true,
                    "term_vector": "with_positions_offsets",
                    "fielddata": {
                        "format": "disabled"
                    },
                    "fields": {
                        "stopped": {
                            "analyzer": "SearchWithSpecials",
                            "type": "string",
                            "index": "analyzed",
                            "store": false,
                            "norms.enabled": true,
                            "term_vector": "with_positions_offsets",
                            "fielddata": {
                                "format": "disabled"
                            }
                        },
...

note that content_cleaned and content_cleaned.stopped should do the same thing but content_cleaned is not searchable for a word, that is found when searching content_cleaned.stopped. So the base default field doesn't work unless repeated within the fields list.

apatrida · 2014-01-22T10:51:48Z

Doing this, does not resolve the problem that content_cleaned does not find words that content_cleaned.stopped finds (although their declaration is the same):

 "content_cleaned": {
                    "type": "string",
                    "index": "analyzed",
                    "analyzer": "SearchWithSpecials",
                    "store": false,
                    "norms.enabled": true,
                    "term_vector": "with_positions_offsets",
                    "fielddata": {
                        "format": "disabled"
                    },
                    "fields": {
                        "content_cleaned": {
                            "analyzer": "SearchWithSpecials",
                            "type": "string",
                            "index": "analyzed",
                            "store": false,
                            "norms.enabled": true,
                            "term_vector": "with_positions_offsets",
                            "fielddata": {
                                "format": "disabled"
                            }
                        },
                        "stopped": {
                            "analyzer": "SearchWithSpecials",
                            "type": "string",
                            "index": "analyzed",
                            "store": false,
                            "norms.enabled": true,
                            "term_vector": "with_positions_offsets",
                            "fielddata": {
                                "format": "disabled"
                            }
                        },
...

martijnvg · 2014-01-22T10:59:35Z

I'm lost on what you're trying to achieve. Can you share a full example (via gist or something like that), with the mapping you're creating, index a sample document and search requests you expect to work.

apatrida · 2014-01-22T11:13:42Z

I figured it out, had a type specific override breaking the mappings that I changed, had to change it as well. All good, wrote a test case that covered the normal usage and it worked fine.

s1monw · 2014-01-22T11:15:49Z

if you wanna contribute that testcase that is always welcome as well! just open a PR

apatrida · 2014-01-22T11:21:11Z

Of course, thanks Simon

This was referenced Dec 19, 2013

Path, index_name, multi_fields and pooled fields #4131

Closed

Custom _all fields #4520

Closed

ghost assigned martijnvg Dec 23, 2013

martijnvg mentioned this issue Dec 29, 2013

Replaced multi-field type with multi-field syntax. #4566

Closed

imotov mentioned this issue Jan 8, 2014

Support multifield path per field #4099

Closed

martijnvg closed this as completed Jan 14, 2014

johtani mentioned this issue Jan 28, 2014

Change es version to 1.0.0.RC1 codelibs/elasticsearch-solr-api#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve multi_field syntax #4521

Improve multi_field syntax #4521

clintongormley commented Dec 19, 2013

clintongormley commented Jan 14, 2014

martijnvg commented Jan 14, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

s1monw commented Jan 22, 2014

apatrida commented Jan 22, 2014

Improve multi_field syntax #4521

Improve multi_field syntax #4521

Comments

clintongormley commented Dec 19, 2013

Proposed syntax for multi_field

clintongormley commented Jan 14, 2014

martijnvg commented Jan 14, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

apatrida commented Jan 22, 2014

martijnvg commented Jan 22, 2014

apatrida commented Jan 22, 2014

s1monw commented Jan 22, 2014

apatrida commented Jan 22, 2014